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Tests  of  an  Improved  Oceanographic  Expert  System 


Introduction 

The  oceanographic  expert  system  (OES)  developed  by  the  Naval  Research  Laboratory's  Re¬ 
mote  Sensing  Applications  Branch  models  some  aspects  of  the  kinematics  of  Gulf  Stream 
mesoscale  features.  As  part  of  a  suite  of  tools  to  assist  the  satellite  image  analyst,  it  models  the 
motion  of  warm-core  and  cold-core  rings  (WCRs  and  CCRs),  and,  to  a  lesser  extent,  the  motion  of 
the  Gulf  Stream  (Thomason  and  Blake  1986;  Thomason  1989). 

The  OES  uses  a  rule  base  to  provide  location-dependent  ring-motion  forecasts.  Motions 
hypothesized  by  the  OES  are  further  modified  by  ring-Gulf  Stream  interactions.  Consequently,  the 
OES  also  requires  the  Gulf  Stream's  position  as  a  function  of  time.  Originally,  the  OES  used  a  sim¬ 
ple  model  of  Gulf  Stream  motion  in  terms  of  downstream  propagation  of  meanders.  Now  it  uses  a 
neural-network-based  forecast  module,  which  provides  better  motion  estimates. 

The  OES  performed  well  in  comparison  with  conventional  numerical  models  (Lybanon  and 
Thompson  1991),  but  development  has  continued.  One  change  improved  the  geometry  used  to 
describe  ring  motion.  A  "natural  language"  explanation  facility  permits  the  user  to  interrogate  the 
system  to  find  out  why  it  projected  a  particular  motion  (Bridges  1992;  Bridges  and  Lybanon 
1993).  A  recent  conversion  into  a  C-based  language  is  intended  to  make  the  OES  easier  to 
implement  on  a  variety  of  platforms  (Bridges  and  Chen  1994). 


Expert  System 

The  OES's  domain  is  the  northwest  Atlantic  Ocean.  That  domain  is  divided  into  nine 
geographic  regions.  In  each  region  the  OES  "moves"  rings  in  the  same  fashion.  The  basic  motion 
is  at  a  specific  speed  in  a  specific  direction;  those  values  differ  from  one  region  to  another  and  for 
WCRs  and  CCRs.  If  a  ring  is  "close"  to  the  Gulf  Stream,  then  the  ring  is  assumed  to  be 
interacting  with  the  Gulf  Stream  and  its  motion  is  modified.  The  closeness  criterion  and  the  details 
of  the  interacting  motion  also  differ  from  one  region  to  another  and  for  WCRs  and  CCRs.  This 
characteristic  of  the  OES  allows  the  ring  motion  rules  to  be  general,  with  the  details  specified  by 
values  in  a  "region  parameters"  file.  As  a  consequence,  changes  to  die  way  the  OES  moves  rings 
can  be  made  simply  by  changing  region  parameters.  That  feature  allows  improvements  to  be  made 
without  the  necessity  of  recompiling  (Lybanon  1990). 

The  region  parameters  file  contains  a  set  of  values  for  each  of  the  regions  and  for  each  type 
of  ring,  warm  and  cold.  Thus,  there  are  18  sets  of  values.  Each  set  includes  an  identifier,  a  speed 
(for  noninteracting  ring  motion),  a  number  from  1  to  16  indicating  general  direction  (used  by  the 
interacting  ring  motion  rules),  a  compass  heading,  a  size  decay  factor  for  noninteracting  rings,  a 
minimum  radius  below  which  a  ring  is  assumed  to  coalesce  with  the  Gulf  Stream,  and  a  decay  fac¬ 
tor  for  interacting  rings.  In  addition  there  are  "breakpoints,"  three  each  for  warm  rings  and  four 
each  for  cold  rings,  used  by  the  interacting  ring  rules. 

The  simplest  change  to  the  region  parameters  file  is  "global"  improvement,  designed  to  min¬ 
imize  the  average  (over  all  WCRs  and  over  all  CCRs)  errors  in  forecast  ring  position.  The  changes 
were  based  on  tests,  in  which  ring  positions  forecast  by  the  OES  were  compared  with  the  observed 
positions  of  the  same  rings  at  the  later  times.  The  tests  and  the  error  measures  are  discussed  in 
greater  detail  in  the  next  section.  The  global  improvement  consisted  of  one  vector  change  in  ring 
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velocity  for  all  WCR  regions  and  a  similar  change  for  all  CCR  regions.  The  speed  and  compass 
heading  parameters  are  the  parameters  that  were  changed.  The  purpose  of  the  change  was  to 
remove  linear  trends  in  mean  vector  position  errors.  Only  noninteracting  (with  the  Gulf  Stream) 
rings  were  considered,  and  only  the  "noninteracting"  region  parameters  were  changed.  The 
Results  section  of  this  report  compares  the  results  obtained  for  both  the  original  ("prototype")  and 
improved  (designated  "Mod  5")  OES  versions. 

Several  changes  were  made  in  the  OES  itself,  which  has  been  under  development  for  8 
years.  Earlier  versions  of  the  OES,  and  its  performance,  are  described  in  the  references 
(Thomason  and  Blake  1986;  Lybanon  1990;  Lybanon  and  Thompson  1991).  As  pointed  out  in  the 
Introduction,  the  changes  included  improvements  in  the  ring  motion  geometry  equations,  the 
addition  of  a  natural-language  explanation  facility,  replacement  of  the  Gulf  Stream  motion  logic, 
and  ongoing  conversion  into  a  C-based  language. 

The  original  OES  did  not  distinguish  between  the  different  sizes  of  a  degree  of  latitude  and 
a  degree  of  longitude,  nor  did  it  account  for  the  variation  with  latitude  of  the  latter  size.  To  first 
order  that  is  an  unimportant  error.  Since  the  latitude  extent  of  a  single  region  is  small,  the  size  of  a 
degree  of  longitude  is  nearly  constant  within  a  region  and  the  error  can  be  compensated  to  a  good 
approximation  by  properly  ("improperly"  more  accurately  describes  the  situation)  defining  the 
region  parameters.  For  this  reason,  it  was  possible  to  get  reasonably  good  performance  from  the 
OES.  However,  the  error  made  it  difficult  to  calculate  and  apply  the  global  improvement 
corrections  described  above.  Also,  the  same  distance  calculation  was  used  to  find  the  distance 
between  a  ring  and  the  Gulf  Stream.  Since  this  distance  is  used  to  determine  whether  a  ring  is 
interacting  with  the  Gulf  Stream,  that  decision  was  erroneous  in  some  cases.  The  geometry  error 
has  been  corrected  both  in  the  (noninteracting)  ring-motion  rules  and  in  distance  calculations. 
Bridges  (1992)  gives  details  of  the  changes. 

The  explanation  capability  was  added  to  give  users  a  basis  forjudging  the  quality  of  the  sys¬ 
tem's  decision-making  process.  It  also  makes  it  easy  to  distinguish  between  interacting  and  nonin¬ 
teracting  rings,  and  to  obtain  other  information  used  in  the  evaluation  tests.  The  explanation  com¬ 
ponent  consists  of  an  introspection  module  and  a  presentation  module.  The  former  "watches”  the 
reasoning  process  and  records  the  data  that  caused  each  rule  to  fire  and  the  new  information 
produced  as  a  result  of  each  rule  firing.  The  latter  can  use  this  information  to  present  a  detailed  nat¬ 
ural  language  trace  of  the  rules  that  have  fired  or  a  shorter  natural  language  summary  of  the  reason¬ 
ing  used  for  the  prediction.  The  structure  of  the  original  OES  was  not  amenable  to  the  incorpora¬ 
tion  of  an  explanation  facility  because  the  knowledge  needed  for  explanation  was  not  explicitly 
represented  in  the  knowledge  base.  So,  prior  to  implementing  the  explanation  facility,  the  structure 
of  the  rule  base  was  revised  with  the  knowledge  "chunks"  in  each  rule  at  a  finer  level  of 
granularity,  and  the  results  of  each  decision  explicitly  assened  into  the  working  memory. 
Although  the  new  rule  base  actually  contains  more  rules  than  the  old  one,  the  rules  are  more 
general.  Not  only  will  possible  future  modifications  be  facilitated,  but  the  system's  speed 
increased  slightly  (Bridges  1992;  Bridges  and  Lybanon  1993). 

The  simple  Gulf  Stream  motion  module  was  replaced  by  an  improved  version  that  uses  a 
neural  network  to  forecast  changes  in  Gulf  Stream  shape  (Chase  and  Holyer  1993).  Another 
change  will  make  it  easier  to  "port"  the  OES  to  other  Navy  facilities,  e.g.  as  a  part  of  the  Tactical 
Environmental  Support  System  (Lybanon  1992).  Hie  OES  was  originally  written  in  a  combination 
of  OPS83  (a  specialized  expert  system  language),  C,  and  Fortran.  The  modifications  described 
above  were  made  in  those  languages,  and  the  resulting  modified  OES  was  used  in  the  tests 
described  in  the  next  section.  However,  a  recent  effort  converted  the  rule-based  portion  of  the 
OES  into  the  C  Language  Integrated  Production  System  (CLIPS,  an  expert  system  tool  developed 
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by  NASA)  rad  die  procedural  OPS83  code  into  C.  The  converted  system  will  be  able  to  run  cm 
any  computer  that  has  a  C  compiler,  and  it  will  be  functionally  identical  to  the  OPS83  version  (bar¬ 
ring  any  future  modifications). 


Results 

The  geometry  changes  guaranteed  that  the  OES  would  give  different  hypothesized  ring 
positions  than  the  previous  versions,  so  it  was  necessary  to  "tune"  the  system.  This  was  done  by 
reverting  to  the  "prototype"  region  parameters,  testing  the  modified  OES,  and  using  the  "global 
improvement"  technique  to  determine  a  new  region  parameter  set 

Tests  consisted  of  comparing  OES  forecasts  for  7  and  14  days  with  observed  ring  positions 
and  sizes  at  the  same  later  times.  Analysis  used  several  error  measures:  "goodness-of-fit" 
measures  for  both  position  and  size,  the  number  of  cases  for  which  the  OES's  position  forecasts 
were  more  accurate  than  a  no-motion  assumption,  and  vector  and  scalar  position  errors. 

The  goodness-of-fit  measure  for  ring  position  is  simply  the  ratio  of  the  error  in  the  forecast 
ring  position  to  the  total  distance  moved  by  the  ring  in  the  time  interval  (7  or  14  days  in  these 
tests),  or 


GOFtr.nd.tion  =  (position  error)  /  (distance  moved). 

Similarly  the  goodness-of-fit  measure  for  ring  sire  is  the  absolute  value  of  the  fractional  error  in 
ring  size  for  the  later  time,  or 


GOFgiie  =  II  -  rforecut  /  r actual!. 


In  both  cases,  small  values  indicate  accurate  forecasts.  Zero  values  mean  a  perfect  match  between 
the  forecast  position  or  size  and  the  observed  value  for  that  time.  That  observation  applied  to 
GOFtnmsiation  leads  to  the  definition  of  another  error  measure. 

GOFtruuiition  =  0  indicates  a  perfectly  accurate  position  forecast.  A  no-motion  forecast 
translates  into  GOF^.i.,;™  =  1,  since  in  this  case  the  distance  the  ring  actually  moves  is  the  error 
in  forecast  position.  So  it  is  clear  that  GOF^.;.^  values  less  than  1  correspond  to  forecasts  that 
are  more  accurate  than  those  given  by  the  no-motion  assumption,  while  values  greater  than  1  cor¬ 
respond  to  less  accurate  forecasts.  Because  of  this,  the  percentage  of  cases  in  which  the  OES's 
forecasts  are  better  than  a  no- motion  assumption  is  easy  to  find. 

One  number  does  not  adequately  characterize  a  data  set,  so  the  analysis  of  test  results 
employed  several  other  error  measures.  One  is  the  mean  value  of  GOFtmuiation-  Position  errors 
themselves  arc  meaningful.  The  numerator  of  GOF translation  is  scalar  position  error,  so  that  quan¬ 
tity  is  immediately  available.  One  might  argue  that  mean  scalar  position  error  overstates  the  error, 
since  all  erf  the  individual  values  are  positive,  regardless  of  direction.  To  take  account  of  this  objec¬ 
tion,  the  mean  vector  position  error  is  also  reported.  This  calculation  resolves  each  error  into  x  and 
y  components,  which  are  separately  averaged  over  the  data  set.  The  resulting  mean  vector  error 
has  the  property  that  errors  in  one  direction  are  compensated  by  errors  in  the  opposite  direction. 
Random  (small,  it  is  hoped)  errors  are  expected  to  average  out,  leaving  only  overall  biases.  So  the 
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combination  of  vector  and  scalar  errors  gives  information  both  on  bias  and  scatter  in  the  position 
errors. 


Figure  1  shows  the  region  parameters  file  with  "prototype"  (i.e.,  those  from  the  original 
version  of  the  system)  values.  Tests  of  the  OGS  using  this  regions  file,  using  the  "Gulfcast"  data 
set  described  in  Lybanon  (1990),  were  evaluated  using  all  of  die  error  measures  described  above. 
Only  noninteracting  ring  cases  went  into  the  statistics.  A  fit  to  the  mean  values  (over  all  non  interact¬ 
ing  WCRs  and  CCRs)  of  the  vector  error  components,  for  7-  and  14-day  comparisons,  yielded 

vector  error  velocity  estimates  for  WCRs  and  CCRs.  That  is,  the  linear  trend  coefficients  for  Ax 

and  Ay  are  effectively  the  components  of  the  velocity  with  which  the  OES  forecasts  deviate,  on  the 
average,  from  ground  truth.  The  global  correction  consisted  of  using  these  velocities  to  correct  the 
speed  and  compass  heading  parameters,  yielding  the  Mod  5  region  parameters  Hie  shown  in  Fig. 
2.  This  "Mod  5  system"  was  tested  against  the  same  data  set. 

Table  1  shows  the  statistics  that  result  from  the  tests  of  both  the  prototype  (upper  part  of 
table)  and  Mod  S  (lower  part  of  table)  versions  of  the  OES.  In  all  cases  the  ring-size  results  show 
that  the  OES  (both  versions)  predicts  ring  size  accurately.  The  ring-motion  results  are  a  different 
matter.  The  prototype  system  achieved  better  results  for  CCRs  than  WCRs.  For  CCRs,  the  per¬ 
cent  better  than  no  motion  statistics  were  higher  than  50%  for  both  7  and  14  days,  and  the  mean 
G OF trant ution  statistics  were  less  than  1.0  for  both  7  and  14  days;  the  opposite  was  true  for 
WCRs.  The  scalar  and  vector  error  magnitudes  of  mean  position  error  were  better  for  CCRs  than 
for  WCRs,  also.  However,  the  Mod  5  system  outperformed  the  prototype  system  with  respect  to 
every  single  one  of  the  motion  error  statistics.  In  particular,  the  vector  error  magnitude  values 
were  drastically  reduced.  This  is  not  surprising,  since  the  global  correction  was  designed  to  accom¬ 
plish  exactly  this  result  The  Mod  5  ring-size  error  statistics  were  very  similar  to  those  for  the  pro¬ 
totype  system.  While  the  greatest  improvement  took  place  in  the  WCR  ring-motion  statistics,  the 
CCR  ring-motion  statistics  were  also  significantly  improved. 

Comparison  with  previous  results  (Lybanon  and  Thompson  1991)  shows  that  the  OES 
with  the  "wrong"  geometry  apparently  obtained  better  results,  at  least  with  respect  to  some  of  the 
error  measures,  for  both  WCRs  (with  Mod  4  parameters)  and  CCRs  (with  Mod  3  parameters). 
Why  then  should  we  not  restore  the  previous  motion  geometry  and  use  the  parameters  that  give 
superior  performance,  despite  the  fact  that  (theoretically,  at  least)  the  old  motion  geometry  was 
wrong?  This  procedure  is  rejected  on  two  counts.  First,  the  old  results  are  not  significantly  better 
than  die  new  results  (and  are  sometimes  worse),  especially  considering  the  small  size  of  the  test 
data  set  (It  may  even  be  true  that  the  old  results  appear  better  than  they  really  are  because  some 
noninteracting  rings  were  incorrectly  labeled  as  interacting;  hence,  they  were  mistakenly  eliminated 
from  consideration.)  Second,  the  old  results  came  after  several  stages  of  improvement,  whereas 
the  results  tabulated  in  Table  1  came  after  only  one  stage  of  improvement.  It  is  reasonable  to 
assume  that  further  improvement  is  possible,  and  the  benefits  of  using  the  correct  motion  geometry 
are  judged  to  outweigh  the  apparent  benefit  of  the  (apparently)  slightly  better  results  obtained  prev¬ 
iously. 


Conclusion 

The  work  presented  here  shows  that  the  expen  system  is  "trainable."  The  modifications 
made  so  far  amount  to  first-order  corrections.  The  Mod  5  changes  to  region  parameters  effectively 
remove  the  linear  trend  from  both  WCR  and  CCR  motion  errors,  and  the  paths  predicted  by  the 
expen  system  are  superior  to  a  persistence  (no  motion)  assumption  75%  or  more  of  the  time  for 
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both  WCRs  and  CCRs,  for  both  7-day  and  14-day  forecasts.  The  mean  scalar  position  errors, 
which  give  a  measure  of  scatter  of  the  forecast  positions  about  the  true  paths,  are  of  order  20  km 
for  14-day  forecasts,  and  less  for  7-day  forecasts.  The  expert  system  shows  promise  as  a  natural 
complement  to  a  foil  numerical  model  formulation  in  the  operational  forecasting  of  ring  motion. 
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WREG1 

8.0 

1 

0.0 

0.5 

100.  0.5 

.9 

.75 

.5 

WREG2 

8.0 

9 

210.0 

0.9 

30.  0.8 

.9 

.75 

.5 

WREG3 

8.0 

10 

240.0 

0.95 

30.  0.85 

.9 

.75 

.5 

WREG4 

6.0 

12 

260.0 

0.99 

35.  0.96 

.9 

.75 

.5 

WREG5 

7.5 

11 

244.0 

0.998 

30.  0.98 

.9 

.75 

.5 

WREG6 

6.0 

12 

249.0 

0.998 

30.  0.98 

.9 

.75 

.5 

WRBG7 

5.0 

13 

264.0 

0.999 

30.  0.98 

.9 

.75 

.5 

WREG8 

5.0 

13 

267.0 

0.999 

35.  0.98 

.9 

.75 

.5 

WREG9 

5.0 

13 

268.0 

0.999 

35.  0.98 

.9 

.75 

.5 

CREG1 

6.0 

10 

267.0 

0.998 

25.  0.8 

0.3 

0.31 

0.8 

0. 

CREG2 

6.0 

10 

249.0 

0.998 

25.  0.9 

0.3 

0.31 

0.8 

0. 

CREG3 

6.0 

7 

249.0 

0.998 

25.  0.9 

0.3 

0.31 

0.8 

0. 

CREG4 

6.0 

10 

260.0 

0.999 

25.  0.95 

0.3 

0.31 

0.8 

0. 

CBEG5 

5.0 

11 

264.0 

0.999 

25.  0.95 

0.3 

0.31 

0.8 

0. 

CREG6 

5.0 

11 

264.0 

0.999 

30.  0.95 

0.3 

0.31 

0.8 

0. 

CREG7 

5.0 

11 

264.0 

0.9999  35.  0.98 

0.3 

0.31 

0.8 

0. 

CKEG8 

5.0 

12 

264.0 

0.9999  35.  0.98 

0.75  0.751 

0.9 

0. 

CREG9 

5.0 

13 

264.0 

0.9999  35.  0.98 

0.75  0.751 

0.95 

0. 

Modified  by  Susan  Bridges  7/31/91.  The  lat  adjust  and  long  adjust  factors 
described  below  have  been  replaced  by  carve  heading  value  that  corresponds  to  a 
ccnpass  heading. 

Items  per  line  are  region  name,  speed  (cm/ sec) ,  direction  (one  of  16  for 
general  reference),  lat  adjust  factor*,  long  adjust*,  decay  when  not  in 
GSinteract,  min  radius  for  no  coalesce  with  GS,  decay  in  GSinteract. 

For  wcregions,  also  breakpoints  bl-3  for  ratio  tests  in  GS  interaction. 

For  ccregions,  also  breakpoints  bl-4  for  ratio  tests  for  looping  effects. 

Further  modified  by  Matthew  Lybanon  2/2/93  to  correct  seme  minor  apparent 
discrepancies. 


*These  are  the  2  quantities  that  have  been  replaced  by  ccnpass  heading. 


Figure  1.  Region  parameters  file  with  "prototype"  parameter  values. 
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WREG1 

10.1 

1 

20 

0.5 

100.  0. 

.9  .75  .5 

WREG2 

5.51 

9 

186 

0.9 

30.  0.8 

.9  .75  .5 

WKEG3 

4.32 

10 

234 

0.95 

30.  0.85 

.9  .75  .5 

WREG4 

2.50 

12 

280 

0.99 

35.  0.96 

.9  .75  .5 

WREG5 

3.77 

11 

241 

0.998 

30.  0.98 

.9  .75  .5 

WREG6 

2.27 

12 

252 

0.998 

30.  0.98 

.9  .75  .5 

WREG7 

1.79 

13 

301 

0.999 

30.  0.98 

.9  .75  .5 

WREG8 

1.95 

13 

307 

0.999 

35.  0.98 

.9  .75  .5 

WREG9 

2.01 

13 

309 

0.999 

35.  0.98 

.9  .75  .5 

CREG1 

4.21 

10 

269 

0.998 

25.  0.8 

0.3  0.31 

0.8 

0. 

CPEG2 

4.28 

10 

243 

0.998 

25.  0.9 

0.3  0.31 

0.8 

0. 

CREG3 

4.28 

7 

243 

0.998 

25.  0.9 

0.3  0.31 

0.8 

0. 

CREG4 

4.21 

10 

259 

0.999 

25.  0.95 

0.3  0.31 

0.8 

0. 

CREG5 

3.21 

11 

265 

0.999 

25.  0.95 

0.3  0.31 

0.8 

0. 

CREG6 

3.21 

11 

265 

0.999 

30.  0.95 

0.3  0.31 

0.8 

0. 

CFEG7 

3.21 

11 

265 

0.9999 

35.  0.98 

0.3  0.31 

0.8 

0. 

CREG8 

3.21 

12 

265 

0.9999 

35.  0.98 

0.75  0.751 

0.9 

0. 

CREG9 

3.21 

13 

265 

0.9999 

35.  0.98 

0.75  0.751 

0.95 

0. 

This  file  was  prepared  10/19/93. 

Items  per  line  are  region  name,  speed  (cm/ sec),  direction  (one  of  16  for 
general  reference) ,  heading,  decay  when  not  in  GSinteract,  min  radius  for  no 
coalesce  with  GS,  decay  in  GSinteract. 

For  wcregions,  also  breakpoints  bl-3  for  ratio  tests  in  GS  interaction. 

For  ccregions,  also  breakpoints  bl-4  for  ratio  tests  for  looping  effects. 

This  version  of  REGIONS . DAT  contains  "mod  5"  changes.  These  are  chosen  to 
remove  the  global  linear  trends  from  the  results  of  applying  the  "prototype" 
version  to  noninteracting  (with  the  Gulf  Stream)  rings  only,  after  Susan 
Bridges  put  the  motion  geometry  corrections  into  the  expert  system.  The 
corrections  came  from  a  2-parameter  linear  fit  to  the  errors. 


Figure  2.  Region  parameters  file  with  "Mod  5”  parameter  values. 


7 


Table  1.  Prototype  and  Mod  5  System  Test  Statistics 

! 


Number 

Percent 

Mean  GOF 

Mean  Pos.  Errors 

of 

better  than 

Vector 

Cays  WCR/CCR  Rings 

no  motion 

size  Translation 

Scalar 

Magnitude 

Prototype  System  Results 

7 

WCR 

23 

40.91% 

0.011 

1.729 

24.768 

21.64 

CCR 

28 

60.71% 

0.021 

0.941 

23.812 

10.15 

total 

51 

52.00% 

14 

WCR 

9 

22.22% 

0.032 

1.896 

48.225 

44.23 

CCR 

8 

75.00% 

0.012 

0.698 

30.45 

20.85 

total 

17 

47.06% 

Mod  5  System  Results 

7 

WCR 

22 

85.71% 

0.012 

0.784 

13.244 

1.71 

CCR 

27 

77.78% 

0.022 

0.603 

19.804 

2.87 

total 

49 

81.25% 

14 

WCR 

8 

75.00% 

0.029 

0.752 

22.181 

4.96 

CCR 

9 

88.89% 

0.024 

0.415 

20.832 

2.5 

total 

17 

82.35% 

GOF  =  goodness  of  fit  Position  errors  in  km 
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